Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[fix](multi-catalog)fix all avro,hudi and jdbc scanner class loader dependencies conflicts #23015

Merged
merged 1 commit into from
Aug 20, 2023
Merged

[fix](multi-catalog)fix all avro,hudi and jdbc scanner class loader dependencies conflicts #23015

merged 1 commit into from
Aug 20, 2023

Conversation

wsjz
Copy link
Contributor

@wsjz wsjz commented Aug 15, 2023 •
edited
Loading

Proposed changes

add preload-extensions module, put all conflict dependencies to pom.xml in preload-extensions

Further comments

If this is a relatively large or complex change, kick off the discussion at [email protected] by explaining why you chose the solution you did and what alternatives you considered, etc...

@wsjz wsjz force-pushed the fix_avro_jdbc_dep branch 3 times, most recently from 1d92b7b to 5b84c80 Compare August 15, 2023 13:09
@wsjz wsjz mentioned this pull request Aug 17, 2023
Closed
@wsjz wsjz changed the title [fix](multi-catalog)fix avro and jdbc scanner dependency [fix](multi-catalog)fix all avro,hudi and jdbc scanner class loader dependency conflicts Aug 17, 2023
@wsjz wsjz changed the title [fix](multi-catalog)fix all avro,hudi and jdbc scanner class loader dependency conflicts [fix](multi-catalog)fix all avro,hudi and jdbc scanner class loader dependencies conflicts Aug 17, 2023
AshinGau
AshinGau reviewed Aug 17, 2023
View reviewed changes
fe/pom.xml Outdated Show resolved Hide resolved
fe/be-java-extensions/java-udf/pom.xml Outdated Show resolved Hide resolved
fe/be-java-extensions/java-common/pom.xml Outdated Show resolved Hide resolved
fe/be-java-extensions/avro-scanner/pom.xml Show resolved Hide resolved
@wsjz wsjz force-pushed the fix_avro_jdbc_dep branch 2 times, most recently from 79b60f5 to 59a38c9 Compare August 18, 2023 03:26
@wsjz
Copy link
Contributor Author

wsjz commented Aug 18, 2023

run buildall

@wsjz
Copy link
Contributor Author

wsjz commented Aug 18, 2023

run external

@wsjz wsjz marked this pull request as ready for review August 18, 2023 09:14
@hello-stephen
Copy link
Contributor

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.95 seconds
stream load tsv: 542 seconds loaded 74807831229 Bytes, about 131 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 30 seconds loaded 861443392 Bytes, about 27 MB/s
insert into select: 29.1 seconds inserted 10000000 Rows, about 343K ops/s
storage size: 17162015008 Bytes

@wsjz
Copy link
Contributor Author

wsjz commented Aug 18, 2023

run external

@wsjz
Copy link
Contributor Author

wsjz commented Aug 18, 2023

run buildall

@hello-stephen
Copy link
Contributor

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.21 seconds
stream load tsv: 535 seconds loaded 74807831229 Bytes, about 133 MB/s
stream load json: 21 seconds loaded 2358488459 Bytes, about 107 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
insert into select: 29.3 seconds inserted 10000000 Rows, about 341K ops/s
storage size: 17161912511 Bytes

@github-actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@zy-kkk
Copy link
Member

zy-kkk commented Aug 19, 2023

run buildall

@hello-stephen
Copy link
Contributor

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 44.83 seconds
stream load tsv: 539 seconds loaded 74807831229 Bytes, about 132 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
insert into select: 29.5 seconds inserted 10000000 Rows, about 338K ops/s
storage size: 17162330278 Bytes

@wsjz
Copy link
Contributor Author

wsjz commented Aug 19, 2023

run p0

zy-kkk
zy-kkk approved these changes Aug 19, 2023
View reviewed changes
Copy link
Member

@zy-kkk zy-kkk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Aug 19, 2023
@github-actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@morningman morningman merged commit 5ba505e into apache:master Aug 20, 2023
16 of 17 checks passed
xiaokang pushed a commit that referenced this pull request Aug 21, 2023
@wsjz @xiaokang
[fix](multi-catalog)fix avro and jdbc scanner dependency (#23015)
133683c 
add preload-extensions module, put all conflict dependencies to pom.xml in preload-extensions
@xiaokang xiaokang mentioned this pull request Aug 30, 2023
Open
@wsjz wsjz deleted the fix_avro_jdbc_dep branch March 28, 2024 07:03
@hidataplus
Copy link

what diffrents about "fe/fe-core/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java" and "fe/be-java-extensions/preload-extensions/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java",???

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@AshinGau AshinGau AshinGau left review comments

@zy-kkk zy-kkk zy-kkk approved these changes

@CalvinKirs CalvinKirs CalvinKirs approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/2.0.1-merged p0_b reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
8 participants
@wsjz @hello-stephen @zy-kkk @hidataplus @CalvinKirs @AshinGau @xiaokang @morningman